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Preface 


This report is written for: 

• institutional leaders responsible for quality in online, open and flexible higher education 

• faculty wanting to have an overview of the field 

• newcomers that want to develop quality schemes 

• policy makers in governments, agencies and organisations 

• major educational stakeholders in the international community 

It is a must read for any person concerned with quality in online, open and flexible higher education. 

The report provides the first global overview of quality models in online and open education, an 
overview which is very timely, delivered as it is for Global Education 2030, the new global educational 
agenda which replaces Education For All, EFA. 

The report paints with a broad brush the landscape of quality in online and open education - and its 
challenges. Illustrating that quality in online learning is as complex as the reality of online learning itself. 
It addresses new needs such as quality in MOOCs and Open Education Resources. It shows that one 
size does not fit all, that improving quality of student experiences is more than ever extremely important, 
and it warns against implementation of quality models that restrict innovation and change. These are all 
important issues to reflect on and discuss. 

It delivers insight into the quality concept, the aspects of quality, and describes a selected number of 
models in relation to certification, benchmarking, accreditation and advisory frameworks, and can 
therefore serve as a guide and inspiration for building quality frameworks. 

While its findings on the one hand shows there is no need for new quality schemes as such, it reveals a 

huge gap and need for knowledge building, knowledge sharing, capacity building and for 
coordination among stakeholders. 

The research team makes 11 recommendations, spanning from important principles such as 
mainstreaming e-learning quality into traditional institutional quality assurance, to topical issues such as 
the establishment of quality criteria for mobile learning systems, and addressing unbundling and the 
emergence of non-traditional providers. Some key recommendations relate to knowledge building and 
sharing, to ensure knowledge resources for guidance and capacity building among experts and 
stakeholders. 

While It is difficult to pin-point one recommendation as the most important or most urgent, my overall 
impression is that its findings - and recommendations on the need for information and knowledge 
sharing, collaboration and coordination are the most crucial and most urgent to address. This 
major and important task can best be carried out in partnership between key stakeholders; inter- 
governmental organisations (e.g. UNESCO, Commonwealth of Learning), quality assurance networks 
(e.g. INQAAHE) and networks of higher education institutions (e.g. ICDE and others). 

The relevance and importance of the work undertaken by the research team can probably best be 
understood in light of the main finding from another recent study: 

“Our results indicate that distance education, when properly planned, designed, and supported by the 
appropriate mix of technology and pedagogy, is equivalent to, or in certain scenarios more effective 

ional face-to-face classroom instruction.” (Kovanovic V, Joksimovic S, Skrypnyk O, Gasevic 

i S and Siemens G (2015) The History and State of Distance Education) 
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Taking note that 414.2 million students will be enrolled in higher education around the world by 2030 - 
an increase from 99.4 million in 2000, and that online, open and flexible education is going mainstream, 

the importance of quality learning outcomes for learners cannot be overestimated. 

I hope that by making this report broadly available, that ICDE contributes to exciting dialogue, 
discussion and development of quality online, open and flexible higher education for the future we 
want. 

Gard Titlestad 
Secretary General, ICDE 
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Executive Summary 

Goals and Project Outcomes 

The global Higher Educational landscape is in a period of dramatic change. Although it is too early to say 
whether these changes will be disruptive, revolutionary or merely evolutionary, a significant driver of 
change has been the dramatic rise in the use and availability of new educational technology. More 
specifically the growth of the Internet is challenging conventional modes of delivery and helping to extend 
access to higher education beyond traditional campus-based learners. In recent years, the demand for 
“online learning”, whether called open, distance, flexible, or e-learning, has grown exponentially in 
response to this new environment Likewise, has the rise of opening up education movement, and the 
growing development with Open Educational Resources (OER) and Massive Open Online Courses 
(MOOC), and the entire unbundling approach in education. Increased internationalisation, widening 
recruitment and upscaling of reaching students are other drivers. Hence, how, where and when students 
learn, how institutions structure programmes and services, and how these services are structured are 
global challenges. Improving quality of student experiences is more than ever extremely important. 

This quality standard study has been undertaken on behalf of The International Council for Open and 
Distance Education (ICDE), a global membership organisation in the field of open and distance education, 
and in formal consultative relations with UNESCO. The study has been conducted by research team 
coordinated by the European Association of Distance Teaching Universities (EADTU). EADTU is 
Europe's leading institutional association in online, open and flexible higher education, and is at the heart 
of the modernisation agenda of European universities. 

The objectives for the study are to establish an overview and analysis of the global situation with regard 
to existing relevant standards and guidelines for open, distance, flexible, and online education, including 
e-learning, encompassing the fundamental notion of students as active participants in an engaging 
learning experience. The ICDE Quality Standard Study 2014 will provide a comprehensive baseline 
study on international quality standards on open and distance learning to underpin further international 
work by ICDE on quality in open and distance learning e.g. actions in collaboration with members, as 
well as with UNESCO and OECD. The report will serve as guidelines for communication, dissemination 
and valorisation activity on quality standards in open and distance learning with stakeholders. 

The study was carried out as desk studies by the researchers, in close collaboration with the international 
Research Advisory Group, ICDE and the ICDE SCOP presidency. The data gathering strategy aimed to 
cover quality standard models, and the discourse on quality in open, distance, flexible and online 
education, including e-learning in all continents trying to show similarities and distinctions due to culture, 
languages and maturity of developing quality. The detailed reviews focused on documents available in 
English with some use of online translation tools to access documents in other languages. The strategy 
aimed likewise to identify a quality spectrum, e.g. certification, accreditation, benchmarking, labelling as 
a frame of reference. The intention was also to address the quality spectrum at macro, meso, and micro 
levels. However, those quality standard models described deeper here in the report are mainly at macro 
and meso level as the literature on impact of quality of individuals’ practice is diffuse. The intention was 
sent the variety of available international quality systems, according to maturity and purpose 
ing and/or enhancing quality in e-learning for institutions and quality assurance bodies. 



Complete report. May 2015 


Page 6 of 52 



INTERNATIONAL 
COUNCIL FOR OPEN AND 
DISTANCE EDUCATION 


Quality models in online and open 
education around the globe: State of 
the art and recommendations 



Lessons Learned 

The review of international quality standard models illustrates that there are many existing schemes and 
models for quality assurance of open, distance, flexible and online education, including e-learning. They 
share many common features and many are designed to offer flexibility for institutions to adapt to suit 
national and institutional contexts. The most common structure encountered presents criteria for 
performance in aspects of institutional management, curriculum design student support and other 
elements of educational provision, further subdivision into performance indicators and indications of 
sources of evidence. The most general categorisation of activities is Management (Institutional strategy, 
visions, and resourcing) Products (processes of curriculum and module development) and Services 
(student, and staff support, information resources etc.). Differences between the models reviewed lie in 
the grouping of criteria and the granularity of the detail applied at the performance indicator levels rather 
than the inherent approach to quality assurance. Some models apply numerical scoring criteria with target 
performance levels others rely on more subjective assessment of performance. There are models that 
require performance assessment of 20-30 items others in excess of 100. The originators of the models 
have each made judgements on the trade-off between generality and specificity in the breakdown of 
activity to be reviewed. Many are designed to integrate with national systems for quality assurance of 
Higher Education that are based on peer review and interrogation of institutional self-assessment 
documents. 

More than forty quality standards models or guidelines from organisations were reviewed. Table 1 below 
summarises features of the most well-known and most used reproduced from the report, and they are 
categorised by their functions and uses: 

Certification/Label is interpreted as a level of recognition granted by the body originating the quality model, 
award of the certificate/label will follow some form of review and may be accompanied by a requirement 
that the reviewed institution commits to an improvement plan and later renewal of certification. The 
originating bodies have various statuses ranging from semi-formal interest groups to international 
representative bodies. 

Benchmarking is a process of comparison of institutional performance with that of others, allocation to the 
benchmarking group indicates that either the originating organisation operates a benchmarking service 
or there is evidence of the model having been used in benchmarking exercises. 

Accreditation is interpreted as a form of mandatory certification or licensing of institutions and/or their 
programmes that grants access to national financial support or recognition of awards for employment 
purposes. Accreditation is a process operated by formal agencies, such as Ministries of Education, 
Quality Assurance Agencies and Professional Bodies. 

Some of the documents reviewed are designed to solely fulfil advisory purposes offering structured 
guidance to the issues associated with open, distance and online education but not presenting processes 
of evaluation or performance measurement (Advisory framework). 

-r,-_ -jhoyys that there is evidence of models being used for multiple purposes and in some instances 
dels and codes developed for Certification/Labelling purposes being formally or informally 
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adopted within national Accreditation processes. There is no shortage of core resource for institutions or 
agencies seeking to formulate to Quality Assurance of Open, Online and Distance and education. 

There are, understandably, more limited sources relating in the areas of OER and particularly MOOCs. 
Some of those that have been reviewed may be considered as derivatives or subsets of earlier systems 
applicable to open, distance and online education, extracting and modifying appropriate criteria from the 
spectrum of Management, Product and Service categories. 

Table 1 Most Used Quality Model (From Table 1 in Ossiannilsson, Williams, Camilleri and Brown 2015). 
The name of the quality model is written in bold and italic then follows the governing organisation, first the 
abbreviation and then the full name spelled out in brackets. In case the organisation doesn't have a 
special model, the organisation is just written with the name. 


AVU (the African Virtual 
University) 

CALED (the Latin American and 
Caribbean Institute for Quality in 
Distance Education) 

CHEA (the Council for Higher 
Education Accreditation), US 



E-xellence 

EADTU (the European 
Association of Distance Teaching 
Universities), NL 

OpenupEd 

EADTU (the European 
Association of Distance Teaching 
Universities), NL 

UNIQUe 

EFQUEL(the European 
Foundation for Quality in e- 
learning), BE 

ECB Check 

EFQUEL (the European 
Foundation for Quality in e- 
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Learning). From Dec 2014 GIZ 
(Deutche Gesellshaft fur 
International Zuzammenarbeit), 
DE 




The eLearning guidelines 
(eLg) 

Ako Aotearoa, developed by 
Tertiary Education Commission, 
led by AUT University and Massey 
University, New Zealand 




The E-Learning Maturity Model 
(eMM) 

New Zealand Ministry of 
Education Tertiary E-Learning 
Research Fund 




E-learning Quality Model (ELQ) 

NAHE (The Swedish National 
Agency for Higher education) 




Epprobate 

The Learning Agency Network 
(LANETO e V), DE 




Khan eight -dimensional e- 
learning framework 

Badrul Khan 




The OLC Quality Scorecard 

Online Learning Consortium, 
(former Sloan-C), US 




OER TIPS 

The Commonwealth Educational 
Media Centre for Asia (CEMCA) 




Pick&Mix 

Matic Media, SERO 
ConsultingLtd, UK 




The review indicates that concepts of quality can be applied at Macro (National/international) Meso 
(institutional) and Micro (individual practice) levels with the formal models reviewed addressing the issues 
at Macro and Meso levels. Less evidence has been found of performance standards at the Micro level, 
but no doubt this will exist within staff development and performance management criteria of those 
institutions that are engaged in quality assurance of their open, distance and online education 
programmes. 

The report explores the concept of the development of maturity in quality assurance processes at both 
institutional and national levels through a progression from quality assurance applied to ensure 
compliance with fixed standards and norms to quality assurance as mechanism for improvement and 
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enhancement and enhancement of provision. The distance education sector with its dependence on 
development of teaching materials, management of scalable systems of student support, etc., has in many 
instances, better developed quality assurance processes than institutions operating traditional 
campus/classroom provision. As conventional institutions and their accrediting agencies seek to adapt to 
the challenges of integrating e-learning and online provision within their quality assurance processes, they 
have much to gain by exchange of experience and approach with the Distance Education sector. 

An extensive selection of quality standard models has been reviewed and analysed. Each one was 
developed for specific purposes, in different contexts, and in different times. It is neither possible nor 
appropriate to recommend one before another, as a selection very much depends on institutional context, 
aim, and maturity. However, the research study identified characteristics, which needs to be addressed 
for quality assurance and quality enhancement. Those can be summarised as below: 

• Multifaceted - e.g. systems use a multiplicity of measures for quality, and will often consider 
strategy, policy, infrastructure, processes, outputs and more so as to come to a well-rounded view 
of holistic quality. 

• Dynamic - e.g. flexibility is built in to systems, to accommodate for rapid-changes in technology, 
as well as social norms. For this reason, they rarely refer to specific technological measures, and 
rather concentrate on the services provided to users through that technology. 

• Mainstreamed - e.g. while all the quality tools surveyed aim at high-level quality improvement, 
this is intended to trickle down throughout the institution and be used as a tool for reflective practice 
by individual members of staff in their daily work. 

• Representative - e.g. quality systems seek to balance the perspectives and demands of various 
interested stakeholders, including students, staff, enterprise, government and society at large. 

• Multifunctional - e.g. most systems serve a triple function of instilling a quality culture within an 
institution, providing a roadmap for future improvement, as well as serving as a label of quality for 
outside perspectives. 

Recommendations to Stakeholders 

From the research study, a set of recommendations were formulated, together with proposed actions for 
stakeholders. The recommendations are: 

1 . Mainstream e-learning quality into traditional institutional quality assurance; 

2. Support the contextualisation of quality systems; 

3. Support professional development, in particular through documentation of best practice and 
exchange of information; 

4. Communicate and promote general principles; 

5. Assist institutions in designing a personalised quality management system; 

6. Address unbundling and the emergence of non-traditional educational providers; 

7. Address quality issues around credentialisation through qualifications frameworks; 

8. Support knowledge transfer from open and distance learning to traditional quality systems; 

9. Support quality assurance audits and benchmarking exercises in the field of online, open, 
flexible, e-learning and distance education; 

10. Encourage, facilitate and support research and scholarship in the field of quality; and 
Encourage, facilitate and support implementing quality assurance related to new modes of 
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teaching. 


From the research study, a set of proposals to ICDE was also suggested, which related to the above 
mentioned recommendations. 

Summarising the survey, analysis and recommendations: 

• There is an extremely large variety of quality tools catering to many audiences and needs. 

• There is no significant gap in terms of analysis of institutional systems, which would require a new 
scheme to be developed. 

• In the case of recognition and unbundling, which are not e-learning specific, there are definite 
deficiencies with scope for further developments. 

• All the quality systems suffer certain deficiencies (lack of universal applicability, unclear which 
maturity levels they are best for, widely divergent quality of reviews and of advice given, challenges 
to respond to change, etc.) 

• There is a role for ICDE working with other international organisations in the following main areas, 
all of which are critical: 

o providing a register of effective quality systems, and a guide to members on which are 
appropriate for their context and purpose 

o addressing common issues around training, best practice sharing, localisation, etc., for 
providers of quality systems 

o working with international organisations to ensure a harmonised regulatory environment 

o working with international agencies to ensure student engagement in determining quality 
standards 
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1 Setting the Scene 

1 .1 Rationale for the Study 

This Quality Standard Study is conducted on behalf of The International Council for Open and Distance 
Education (ICDE), a global membership organisation in the field of open, distance, flexible and online 
education, including e-learning, and in formal consultative relations with UNESCO (see Appendix 2). 

The objectives for this ICDE Quality Standard Study 2014 were to provide an overview of the global state 
of the art regards to existing relevant global guidelines, benchmarks and quality standard models for open, 
distance, flexible, online education, and e-learning, including online post Higher Education, like Massive 
Open Online Courses (MOOC) and Open Educational Resources (OER). The research also aimed to 
analyse stakeholders’ perspectives. The objectives for the study were also to provide and present analysis 
and recommendations to ICDE regarding future work. This research and the report is a contribution and 
can serve as a communication and foundation for policy and strategy visions and missions. The 
background and aim for this Quality Standards Study 201 4 1 is further described in Appendix 3. 

1.1.1 Impact of the study 

The ICDE Quality Standard Study 2014 will provide a comprehensive baseline study on international 
quality standards on Open and Distance Learning to underpin further international work by ICDE on quality 
in Open and Distance Learning, e.g. actions in collaboration with members, as well as with UNESCO and 
OECD. The report will serve as guidelines for communication, dissemination and valorisation on quality 
standards in Open and Distance Learning with stakeholders. 

The impact of the study will serve as guidelines for ICDE and its members both related to required 
resources, but also to ICDE’s current 2013-2016 (Appendix 2), and next Strategic Plan 2017 and onwards 
according to the call (Appendix 3). 

1.1.2 Project approach 

The ICDE Quality Standard Study 2014 was mainly based on desk analysis and previous research work 
by the project team and other international researchers, and through earlier conducted projects in the 
area of quality. Primarily, the secondary sources mentioned in the Joint Bid were used for the mapping 
and classification. Although, an update review of currently assembled resources was undertaken. The 
project team initially established a research advisory group (RAG) with global outreach and with 
organisations and persons who had impact in the areas of quality standards study models. A stakeholder 
identification and analysis exercise was also carried out for the study. 

The data gathering strategy aimed to cover all continents trying to show similarities and distinctions due 
to culture, languages and maturity of developing quality in online learning, including e-learning. The study 
focused mainly on documents available in English language versions. Documents in other languages e.g. 
Spanish, were, in some instances translated, using online translation tools. 

The strategy aimed likewise to identify quality spectrum, e.g. certification, accreditation, benchmarking, 
labelling and frame of references. The purpose of this strategy was to cover quality spectrum at macro, 
meso level, and micro levels. However, those described deeper here in the report are mainly at macro 


http://www.icde.orq/filestore/News/Callforproposal-ICDEqualitvstandardsstudv.pdf 
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and meso level as literature on impact on quality of individuals’ practice is diffuse. The purpose was also 
to present the variety of available international quality systems, according to maturity and purpose for 
measuring and/or enhancing quality in e-learning for institutions and quality assurance bodies. 

The data gathering strategy also aimed to include or at least to discuss and reflect on the emerging 
movement of opening up education like post-traditional online Higher Education as OER and MOOCs and 
likewise. An overview of the methodology and research design is described in Appendix 5. 

1.1.3 Governance of the project and its organisation 

The European Association of Distance Teaching Universities (EADTU) 2 is the contractor for this project. 
EADTU is Europe's leading institutional association in online, open and flexible higher education, and is 
at the heart of the modernisation agenda of European universities (Appendix 1). 

The project lasted between August 2014 and March 2015. The project management approach adopted 
by the project team is broadly based on Prince 2 3 methodology and terminology (Appendix 4). A research 
advisory board (RAG) was established at the beginning of the project, as well as the project assurance 
monitoring group. The project activities have continuously been discussed with the RAG, and the project 
assurance monitoring group, as well as with ICDE. The preliminary results of the research study were 
presented during the ICDE SCOP meeting in November 2014. 

1.1.4 Structure of the report 

The structure of this report consists of five parts. Part 1 describes the project context. In Appendix 1, a 
presentation of the authors is given. Appendix 2 describes briefly ICDE’s Strategic Plan 2013-2016 and 
Appendix 3 reproduces the call for proposals for this study. In Appendix 4, a presentation of the project 
management and governance is given, while in Appendix 5, a glossary is presented. Part 2 is a short 
description of the methodology. A more extended description of the research methodology is given in 
Appendix 6. Part 3 presents the results and findings. In Appendix 7, descriptions are given of the most 
well-known and used quality standard models. In part 4, a discussion and conclusions can be found. In 
part 5, a set of recommendations and proposals are given. The report ends with references and the 
appendices. 

1 .2 Context 

The global landscape of Higher Education is in a period of dramatic change. Some have labelled the 
changes as disruptive, others evolutionary, and some revolutionary. By whatever label: how students 
learn, where and when they learn, how institutions structure programmes and services, and how these 
services are priced and organised are global challenges. A significant driver of the changing landscape 
has been the dramatic rise in the use of technology and, through various modes of delivery, the extension 
of the traditional campus to more learners. The new concept and consequences of unbundling in the 
educational area are drivers as well. Hence, quality issues are more than ever on the educational agenda. 

Globally, Higher Education faces challenges associated with changing social, developmental and 
technological change, and there are many initiatives and reports on modernisation of Higher Education. 

2 http://www.eadtu.eu/about-eadtu/about-eadtu 

best-manaqement-practice.com/qempdf/bmp-qlossary-prince2.pdf 
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In some contexts the challenges relates to changing perceptions of the function and value of Higher 
Education as public or private good resulting from the increased participation rates. In other contexts the 
challenge is of transforming Higher Education systems that have failed to keep pace with increasing 
populations and struggle to provide campus capacity to maintain participation rates let alone increase 
them. In the middle ground, there are education systems that are maturing as national economies develop, 
but where participation in Higher Education is significantly below that of developed western economies. 
Other trends are associated with globalisation of commerce and trade enabled by technological 
developments, affecting both goods and services; result in an increasingly global market for those with 
graduate level qualifications. Thus, not at least according to UNESCO (2015) nations and institutions must 
consider the future directions of what is taught and how it is taught. 4 Policy statements will inevitably refer 
to Quality in Higher Education, but without necessarily defining quality. Few would deny that quality is “a 
good thing”, but views on what constitutes quality are very frequently subjective and subject to context 
specific factors. The rapid spread of digital technologies and the variations in the use patterns based on 
regional infrastructure and social factors means that they provide tools for addressing educational 
challenges, but patterns of use will vary. 

The online education sector, with its history of pioneering new modes of education, is rich in experience 
of many of the factors that are cited as relevant to the challenges of expansion and utilisation of digital 
technologies. In the 1990s, John Daniel, in his book Mega universities and Knowledge Media (Daniel 
1996), challenged the longstanding assumptions of the linkage between exclusivity and excellence in 
Higher Education, citing the structured approaches to module development and student support 
pioneered by the distance education sector. Yet, two decades on quality assurance of the e-learning 
sector is still considered to be a challenge. 

Thus, the online and distance education sector is well accustomed to rising to the challenges now facing 
the education sector and is in a position to contribute strongly and show leadership in many areas. In 
addressing the global challenge three very broad contextual categories can be drawn: 

i) the old developed world of N America, Europe and Australasia 

ii) the newly developed world, South and East Asia, Latin America 

iii) the still developing world of Africa and outliers in Asia and Latin America 

In i) the landscape is characterised by mature higher educational systems, each with a small cluster of 
institutions dating back hundreds of years, a more significant cohort of institutions with origins in the late 
1 9 th century and significant expansion from mid-20 th century onwards. In ii) oldest institutions may typically 
date from mid-19 th century, but with most significant growth being from mid-20 th century onwards. In iii) 
systems with token Higher Education provision in colonial era, significant expansion in early 
decolonisation era, but in more recent decade’s growth failing to meet needs of expanding populations 
and changing technologies. 

Irrespective of regions, the following challenges may be encountered in introducing or adapting quality 
assurance regimes appropriate for current contexts. 


, .utpl.edu.ec/inqles/?q=linkinq/interinstitutional/international-proiect 
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i) Adaptation and minor modification of current systems to remove constraints that unnecessarily 
relate to norms related to traditional face-to-face provision. 

ii) Introduction of credible systems of quality assurance that simultaneously address both face to 
face and online provision. 

iii) Historical patterns of development of distance education that lock in norms and standards 
relevant to a particular set of delivery modes and do not facilitate innovation or introduction of 
new technologies. 

While in all systems face-to-face delivery of teaching remains the predominant mode requirements for 
expansion have led to adoption of technologically assisted teaching systems, exploiting in turn print and 
post, web-based and social networking technologies. The longstanding Open University systems have 
vast experience in delivery and quality assurance of programmes delivered through the technologies of 
the 1980s and 1990s, but their scale may constrain them from easy adaptation to more recent 
technologies; hence there is an urgent need to develop quality assurance processes and performance 
standards for large scale systems based on the low-cost personal devices that may offer the best 
prospects for transformation of for example African Higher Education. 

The recent Tuning Africa report jointly funded by European Union and African Union describes the 
challenges faced by the Higher Education sector and the role of distance learning and ICT. 5 They highlight 
arguments on distance learning, ICT and quality assurance, as follows: 


Distance Learning 

The huge need for access to tertiary education, economies of scale for small countries, the 
growing trend of ephemeral knowledge and the necessity for lifelong learning, flexibility, 
versatility and robust outreach will continue to fuel growth in distance learning. But weak 
quality assurance mechanisms, poor publicity, personnel who are inadequately trained in 
distance education, limited and unreliable ICT access and controversial content dimensions 
confront its success. 

ICT 

While major strides in ICT access have been made, there remain significant institutional, 
infrastructural and technical challenges to institutions making effective use of the 
technologies. 

Quality Assurance 

By 2012, twenty-one African countries had established such agencies and a dozen other 
countries were at relatively advanced stages in moving towards this direction. Francophone 
Africa is lagging; only five such countries in Sub-Saharan Africa have quality assurance 
agencies. 

More than 60% of these quality assurance agencies have been created during the last decade and 
many of them still lack the capacity needed to implement their mandates effectively, necessitating 
capacity building in quality assurance. 


5 www.tuninqafrica.org Report on Tuning Africa project EU/African Union Univ Deusto 2013 
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Recent initiatives by distance education agencies, the African Council for Distance Education (ACDE) and 
African Virtual University (AVU) aim to support institutions. ACDEs development of the COL/DEMP 
framework was finalised in 2012 and has been used for institutional review processes in several of the 
African Open University systems, notably National Open University Nigeria and Open University 
Tanzania. The AVU’s Quality Assurance Framework was published in September 2014. There is much 
common ground between the ACDE and AVU frameworks and it is likely that they will be influential in 
shaping future developments with AVU having particular relevance in Francophone and Lusophone 
countries. 

In Europe, the initiative from the European Commission on Opening Up Education in 2013 aimed to set 
out a framework for enhancing learning and teaching through new technologies and open digital content 
at all levels of education. They argued that how online and open education is changing will have an impact 
on how education is resourced, delivered and taken up. Over the next couple of years, e-learning will 
grow fifteen-fold, and accounting for 30% of all educational provision, thus this transformation should be 
shaped by educators and policymakers, rather than something that simply happens to them. Especially 
in higher education, new technologies act as change agents as they enable universities to meet a broader 
range of learners’ needs, adapting traditional teaching methods and offering a mix of face-to-face and 
online learning possibilities that allow individuals to learn anywhere, anytime. They also create openings 
to engage in new kinds of collaboration and offer opportunities to distribute resources more effectively. 
However, many universities and even governments have been slow to take the lead (EC 2013). Hence, 
the European Commission followed up with a set of recommendations to drive national authorities and 
institutions forward to enhance increased digitization for higher quality and competitiveness in education 
(EC2014). 

However, the overarching paradox is that online and distance education systems with their digital content 
and the persistent record of online transactions provide a rich source of evidence to enable quality 
assurance and audit processes. If open and distance learning were the current dominant mode of Higher 
Education and lecture-based education the innovation, the challenge would lie in how to quality assure a 
form of education in which interactions at the core of the system were ephemeral, highly dependent on 
personal interpretation by the teacher and student and seldom directly monitored. 


1.3 Global Challenges in Open Online Education 

The online education sector with its history of pioneering new modes of education is rich in experience of 
many of the factors that are cited as relevant to the challenges of expansion and utilisation of digital 
technologies. Some rising global challenges on opening up education are outlined as below: 

Scalability: Delivery at scale has been the essence of the founding missions of the many national Open 
University systems established from the 70s onwards. For these institutions, current challenges relate to 
managing transitions to new technologies in a way that does not disrupt well-understood existing system. 
Increased demand for rapid response to enquiries, increased personalisation of study, driven by 
expectations set by online services in the commercial sector, challenge the economies of scale achieved 
by these institutions. 
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Openness: Open, distance, flexible and online education, including e-learning by definition, offers 
openness in location and in most cases also of time of study, but the strong traditions of openness with 
respect to entry qualification give some distance education providers significant experience of non- 
traditional students. Open, distance, flexible and online education, including e-learning providers have 
also been influential in initiation of the Open Educational Resources (OER) movements and the Massive 
Open Online Courses (MOOCS) of both x and c categories. 

Individualisation: Seen by many as the reciprocal of scalability, however through their use of student 
centred resource-based teaching systems, distance education institutions deliver individualised learning 
experiences. Sophistication in their use of Virtual Learning Environment (VLE) systems may enable the 
delivery of individualised programmes more effectively than the transformation of the conventional sector. 
Related to individualisation or personalisation is the entire discourse on accessibility, which also raise a 
lot of questions and demands on how to tailor learning pathways. 

Unbundling and the rise of non-traditional educational providers: Unbundling is a new paradigm in 
the educational sector, hence a longer explanation follows. The separation of the various aspects of 
education, resources, teaching, assessment, etc., is again central to the operations of distance education. 
Open, distance, flexible and online education, including e-learning institutions in creating business models 
appropriate to their context, undertake a thorough analysis of systems requirements and evolve structures 
that may be significantly different from those of conventional institutions. 

Non-traditional educational providers generally offer education that is mostly coursework or modules, with 
a range of courses that could be career-focused or general education or general interest. They often offer 
student support by course assistants rather than academic faculty; students attend episodically and up 
until now providers have relied mainly on market forces for judgements as to their quality. Until recently, 
providers of adult and/or professional training have formed the most prominent part of this group. In 
employment related domains, the certifications offered by IT and software providers are highly valued for 
their immediate applicability in the workplace environment and used to complement or even substitute for 
formal Higher Education qualification. For example, South Asian based providers of Information and 
Technology (IT) training and certifications have expanded both geographically and in the scope of their 
offerings. They provide qualifications that are designed to provide evidence of generic employability skills 
that complement a degree from a standard state university. Likewise, major mobile phone networks are 
adding educational services to their portfolio of customer services. They bring to their activities extensive 
experience of the operation of the quality assurance regimes operated the software and customer service 
industries. 

The partitioning of teaching responsibilities has long been commonplace in open, distance, flexible and 
online education, including e-learning institutions whose scalability is often dependent on the use of locally 
based tutors to interface directly with students. Monitoring the performance of these staff is part and parcel 
of their quality assurance processes. There is widespread evidence, from Europe, Asia and Americas that 
open, distance, flexible and online education, including e-learning providers, satisfy national regulators 
with the quality of their student support services and often rank higher than conventional institutions. 

: MOOCs, and the creation of providers such as Coursera, and others, where institutions and 

i collaborate on course design and provision, has created a significant new category of provider, 
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which falls neither under the category of Higher Educational/Vocational Institutions or of typical adult 
training centres. 

In addition, the open education movement is serving as a catalyst for other types of companies providing 
only specific steps in the process, including: 

• Companies licensing course content produced by educational institutions; 

• Universities who specialise in awarding credit for recognition of prior learning, and supplementing 
it with a few taught credits to acquire a degree; 

• Specialist examination and certification companies; and 

• Textbook publishers who enhance the classroom experience by creating online learning 
experiences and communities to accompany their textbooks. 

Thus open, distance, flexible and online education sector is well accustomed to rising to the challenges 
now facing the broader education sector and is in a position to contribute strongly and show leadership in 
many areas. 
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2 Research methodology 


The issues addressed in this project were the provision of an overview of standards, guidelines and 
benchmarks for quality in open, distance, flexible, and online education, including e-learning, described 
in a systematic and easy understandable way. The analysis of and the recommendation for which 
standards and guidelines that are most relevant for the ICDE membership, should take into account the 
main differences among the ICDE membership throughout the world, e.g. geographical area, state versus 
private institutions, political support for open and distance education, and existing quality structures. 
Furthermore an analysis of opportunities for ICDE should be addressed to align ICDE’s work with that of 
key national and international stakeholders, including quality agencies. Recommendations for ICDE’s 
future work and strategies on quality guidelines, benchmarks, standards and quality should be given 
together with a presentation of a series of proposals, which ICDE may realistically pursue, including an 
analysis of resources required. 

The schedule and resources available did not allow for an extensive interview process, so research was 
based predominantly on readily available documentation and targeted contact with known activist in the 
field of quality. 

There is probably no topic in education which is so discussed and controversial as quality. One discourse 
on quality in the domain of open learning in the 21 st century is, as Ossiannilsson (2012) discussed in her 
research, in the area on quality in e-learning. This is also emphasised by Uvalic-Trumbic and Daniel (2013 
2014), and recently, Bates (2015) argues that quality is defined: 

...as methods that successfully help learners develop the knowledge and skills they will require in a digital 
age . 6 

The concept of a “quality e-learning system” refers to “one in which the learner has a reasonable 
opportunity for success in reaching their learning goals”. 

A glossary is given in Appendix 5. In this report, often the term open online education is used; however, 
the concept refers as well to e-learning, flexible learning, online learning, distance education and distance 
learning, etc. A more comprehensive description of the research methodology is given in Appendix 6. 


'.tonybates.ca^OIS/C^^S/what-do-we-mean-bv-qualitv-when-teachinq-in-a-diqital-aqefflcomments 
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3 Results and Findings 


Throughout the research, there is an evident conflict between use of the word quality as a comparative 
term and its use in the description of systems for the management of institutional processes. A literature 
survey presents an overview of the state of international situation and the subsequent presentation of 
results attempts to provide a pathway through the various interpretations and systems in use. The concept 
of a spectrum of approaches to quality is introduced, as are concepts of progression to maturity by both 
institutions and quality agencies. Key characteristics of quality systems are introduced and stakeholder 
interests and perceptions explored. 

3.1 Literature Survey 

The challenges of quality assurance of e-learning are a longstanding topic of interest with a growing 
literature base for review. Frydenberg (2002) analysed quality dimensions in a number of quality models 
for e-learning, and she proposed nine criteria areas as domains of e-learning quality. They are as follows: 
executive commitment, technology infrastructure, student services, instructional design and course 
development, instruction and instructor services, financial health, program delivery, legal and regulatory 
requirements and program evaluation (in Ossiannilsson 2012 p 65). In a study of quality in online 
education programs Shelton (2011) argued that quality is strongly related to and requires strong and 
ongoing support, motivation and overall policies. In this survey, this was more important than technology, 
cost-efficiency and management (in Ossiannilsson 2012 p 68). Ossiannilsson (2012) investigated and 
reviewed in her research on benchmarking e-learning in Higher Education also international Quality 
Standard models (e.g. available in English). She argues for a conceptual and holistic approach to quality, 
and to see quality as a dynamic process, as the area of e-learning is changing rapidly. Furthermore, she 
argued the importance to raise and create culture of quality at all levels, within the Institution, with staff 
and students, more than to rely on standards set from above. She also emphasised the importance to 
include students in the quality enhancement work and mission. Jung and Latchem (2012) make the 
following important points about general quality assurance processes within institutions, based on a 
review of quality assessment processes in a large number of online and distance education institutions 
around the world. They stress the needs to take a systemic approach to quality assurance, to see quality 
assurance as a process of continuous improvement, and to move the institution from external controls to 
an internal culture of quality. Furthermore they argue that the leading measures of quality focus on 
outcomes. Finally, they state that as poor quality has very high costs, so investment in quality is 
worthwhile. 

ICDE’s 201 1 study of quality assurance regimes in the Asia Pacific region indicated that almost all 
governments in the region had systems in place for the accreditation and quality assurance of Higher 
Education with many, specifically addressing distance education and online provision. While the ICDE 
study comprehensively documents the systems a study funded by the International research Centre 
(IDRC) Openness and Quality in Asian Distance Education, investigated the actions of institutions. It 
covered 16 distance educators’ providers/programmes from a wide range of countries, e.g. India, China, 
Thailand, Indonesia, Sri Lanka, the Philippines, Malaysia, Japan, Singapore, Hong Kong, Korea, Pakistan 
)lia. It shows a variety of approaches taken to develop and improve quality assurance systems 
Education and in addition useful insights on their own quality context have been given (Jung, 



Complete report. May 2015 


Page 20 of 52 


INTERNATIONAL 
COUNCIL FOR OPEN AND 
DISTANCE EDUCATION 


Quality models in online and open 
education around the globe: State of 
the art and recommendations 


Wong & Belawati 2013). Some lessons learned from this survey are that most important is the creation of 
a quality culture. It is difficult to create a culture of quality through just top-down processes. A culture of 
quality can only be implemented and become sustainable when the staffs involved take ownership of the 
processes. Responsibilities for quality need to be situated as close as possible to the people involved, 
and to the operations of given processes. However, it was shown in addition that in countries with national 
quality policies in place, institutions are keener and have stronger potential and motivation to work on 
development of sustainable quality cultures down the line. Furthermore, it was explicit in the survey that 
top institutional leadership had an important role to play encouraging and providing necessary 
infrastructure, resources, training and support, including incentives for staff and students. Similar findings 
were explicit in research by Ossiannilsson (2012) on benchmarking e-learning in Higher Education, 
lessons learned from international projects. 

In the Asian survey (Jung, Wong & Belawati 2013), many institutions reported use of the ISO 9000 system 
demonstrating the applicability of approaches to quality used in the commercial world to Higher Education. 
For example, the Open University of China (OUC) developed a quality assurance framework with 35 
quality indicators across five quality areas: teaching resources development and management; teaching 
process management; learning support services; teaching management, and, teaching and learning 
environment (Du, Yang, Yin, and Zhang 2009, in Jung, Wong & Belawati 2013). Most of the other 
institutions had similar quality indicators for success. 

Important challenges for quality development and quality assurance were raised from the Open University 
of Sri Lanka, they argue that a quality framework should be flexible and dynamic and have the capacity 
to adapt to changing environments. They also identify the importance and challenges of capacity building 
for e-learning to be successful; there are needs for investment in human resources for knowledge, training 
and research in online learning and e-learning. Many academics are not familiar with online learning, 
hence there are huge demands for professional development and institutions should consider needs and 
resources on both staff recruitment, and induction, but also provide in-house training, retraining through 
investment in staff development centres. Other significant problems lie in convincing all staff members 
that quality is a matter of continuous enhancement to achieve excellence and to be sustainable. It is also 
very much about trust and commitment for each individual to develop a culture of quality. This was 
confirmed by several of the other institutions. Overall concluding lessons from the Asian study by Jung, 
Wong & Belawati (2013) were to: 

• Adopt a balanced, systemic approach 

• Focus on pedagogy, learner support and management 

• Move towards a performance and outcomes-based approach 

• Promote a culture of quality and continuous improvement 

A comprehensive review of quality assurance of distance and online education in Latin America 7 identifies 
that the majority of countries in the region have national quality assurance agencies and a growing number 
have specific requirements relating to the provision of distance and online education. A consortium of 
nations and institutions was established to develop quality assurance methodologies and criteria. From 
2003 the CALED guidelines and criteria based around nine major criteria and cascading sub-criteria have 
been used for institutional self-assessment purposes across Latin America (La calidad y los estandares 

7 https://abelsuinq.files.wordpress.com/201 1/05/libro-la-calidad-de-la-educaciocc81n-virtual-virtual-educa- 
uladech.pdf 
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de medicion de la educacion virtual y a distancia en Ecuador: Maria Jose Rubio Gomez p 227). As in 
Latin America there have been barriers to the establishment of large scale unimodal distance education 
institutions the CALED criteria are particularly applicable to the implementation of distance and online 
education in bimodal institutions. The principles of self-assessment and flexibility to tailor the criteria and 
any scoring mechanism to the particulars of institutional purposes renders makes them particularly 
valuable as a regional resource. Integration with other regional systems, such as those developing in 
Europe, is identified as a goal of the CALED organisation. 

The African Council for Distance Education development of the COL/DEMP framework was finalised in 
2012 and has been used for institutional review processes in several of the African Open University 
systems, notably National Open University Nigeria and Open University Tanzania. The recently published 
AVU Quality Assurance Framework (Sept 2014) sets out performance criteria in seven major areas. Six 
of these are commonly encountered in other systems for QA of ODL, but the seventh relates to Community 
Capacity Building, Development and Engagement. These capacity building factors are core to AVU’s 
objectives of facilitating expansion of Higher Education in Africa through provision of resources and 
services to enhance the capacity of existing institutions and agencies. The AVU criteria build on COL 
experience and other examples of international best practice. The Framework is adaptable to specific 
institutional or national contexts. The 92 criteria statements are accompanied by suggested performance 
indicators and evidence sources with a proposed five point score rating associated with each performance 
indicator. There is much common ground, as mentioned above between the ACDE and AVU frameworks 
and it is likely that they will be influential in shaping future developments with AVU having particular 
relevance in Francophone and Lusophone countries. At national level, the Kenyan quality assurance 
agency incorporates sections devoted to online and distance learning within its quality code. 

Recently, the European University Association (EUA 2014) conducted a research survey and mapped the 
state of the art of e-learning in Higher Education in Europe, with responses from 37 countries in EU and 
the wider Europe. The main issues of the survey were on support structures/services, intra-institutional 
infrastructure, coordination, quality assurance and recognition. The research showed that the rationales 
of offering e-learning courses were mainly due to pedagogical and economic motives, demands and 
needs for flexibility in time and place for learners, both for residential students and a wider range of 
professionals and other lifelong learners. Rationales were also better use of resources and to enable 
learners with transversal skills and training, and for entrepreneurial skills. All participants in the survey 
called for the demands of curriculum changes, as well as assessment methods. The research showed in 
addition that most Higher Education institutions were reviewing their e-learning strategies, in order to 
mainstream and to implement e-learning. Additionally, they emphasised that e-learning is a catalyst for 
innovation. The research showed however, that institutions did not pay much attention to mainstreaming 
quality assurance or quality enhancement related to e-learning. 

Nevertheless, the results from the survey emphasised the demands of mainstreaming e-learning and 
hence quality issues and strategies both on national and institutional level, rather than reliance on single 
departments or some enthusiastic teachers and students. The results emphasised demands on 
Institutional governance and management, in addition with both internal and external quality assurance 
and quality enhancement. 


From the European Commission’s report on Modernisation of Higher Education (EC 2014) at least three 
recommendations directly focusing on quality, those three refers to: 
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• The integration of digital technologies and pedagogies should form an integral element of Higher 
Education institutions’ strategies for teaching and learning. Clear goals and objectives should be 
defined and necessary organisational support structures established to drive implementation 
(recommendation 3). 

• All staff teaching in Higher Education institutions should receive training in relevant digital 
technologies and pedagogies as part of initial training and continuous professional development 
(recommendation 5). 

• Governments and Higher Education institutions should work towards full open access of 
educational resources (recommendation 13). 

The concept and consequences of those recommendations were highlighted as lessons learned even in 
the report from Asia mentioned above & Jung, Wong & Belawati (2013). 

According to Uvalic-Trumbic and Daniel (2014), quality assurance of post-traditional Higher Education is 
not straightforward, due to openness and flexibility being primary characteristics of new approaches, as 
traditional approaches to quality assurance are designed for teaching and learning with more tightly 
structured frameworks 

3.2 Quality Concepts 

The term “quality” may be used as a comparator to distinguish a product displaying good attributes from 
one displaying bad, and its use in this way is encountered in higher education through references to quality 
universities by the press and politicians. In this report, the focus is on the use of the term quality in the 
context of the quality management processes used in the provision of education and their deployment 
with the purpose of improving performance from key stakeholder perspectives. The terminology adopted 
aligns closely with that of quality assurance in the business and service sectors and of Quality Codes 
such as ISO 9001. They are dependent on clarity of purpose at organisational level and of personal 
responsibility and accountability at individual level. Thus in organisations using a quality approach clear 
and explicit statements of the organisations’ goals should result in a downward cascade to departmental 
and individual levels so that all can recognise their role in achieving organisational goals. Quality 
assurance audits aim to test this cascade process through examination of documentation and interview 
of staff. The adoption of this concept in higher education is now widespread. 


3.3 Quality Spectrum 

The concept of quality in online education can be elusive and complex. Conversely, educators continue 
to seek out ideal learning environments and share effective practices for advancing quality. Uvalic- 
Trumbic and Daniel (2013 2014) argues that the concept of quality in online learning is as complex as the 
reality of online learning itself. It used to be emphasised that quality is not anything which is, but something 
which is created or caused due to its context. According to Pirsing (1994, p 241, In: Uvalic-Trumbic & 
Daniel 2014) Quality is not a thing, it is something which forms itself in its process. It is an addiction (Italics 
: — — 1\ There j s a saying which states that quality is in the eye of the beholder. Hence, there are 
even focus on what the learners (students) themselves see and define as quality. As 
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consumers/customers their views may not necessarily be the same as other stakeholders. Accordingly 
there are demands for their involvement in co-definition of quality. 8 

3.3.1 Quality on macro, meso and micro levels 

As the concept quality is complex and with a variety of stakeholders quality in e-learning can, according 
to Nordkvelle, Fossland & Nettleland (2013), be reviewed from three levels, e.g. macro, meso and micro 
level. The three levels can in short be described as macro level meaning national/global general 
dimensions, meso level refers to institutional matters, and finally, micro level refers to the course/module 
as such. Hence, when deciding to enhance or review quality in e-learning in Higher Education selection 
of quality model or quality systems to use, will be influenced by the level and aim of the review. The 
majority of systems reviewed operate at the meso level. 

3.3.2 Interpretations of quality concepts 

It is possible to categorise quality assurance systems in Higher Education as presenting a spectrum of 
interpretations of quality concepts, often dependent on national (macro) and institutional (meso) context 
and potentially identify steps on a progression to maturity. 

The concept of organisations progressing through levels of maturity in their internal competences is best 
exemplified in the software industry in which the Software Engineering Institute (SEI) Competency 
Maturity Model’s five levels are used to categorise software development companies. Those levels are: 


• The first level is the initial state (there are no knowledge, nor any enhancement process or quality 
control or costs mechanisms). 

• The second level is the repeatable (there are some knowledge, work is done on repeatable base, 
there are some enhancement processes, and quality and costs can be explicit afterwards). 

• The third level is the defined (this means fully knowledge, there are enhancement processes, and 
quality and costs can be explicit predicted). 

• The fourth level is managed (there are some eligibility, changes and processes can be measured 
and even the impact of enhancement processes can be improved). 

• The fifth and last one is optimised (organisations operate internal quality assurance systems that 
provide full confidence in their ability to identify and rectify systems deficiencies). 

There is no formal equivalent in the quality assurance of Higher Education, but the concept is broadly 
relevant in presenting a progression from quality assurance approaches that focus on compliance with 
standards to those that focus on enhancement. 


8 From a Quality Assurance code perspective there are needs/requirements for students to be consulted in the 
preparation of the institutional self-assessment and engaged in meetings associated with the quality reviews. This 
is what ENQA ESG says and many European systems require. 
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In exploring this journey to maturity it is important to differentiate between norm-based and process- 
based interpretations of what constitutes quality. 


As in other sections of the report we characterise the terminology as: 

Accreditation as being a formal process of recognition or licensing operated by or on behalf of 
a regulating agency. 

Certification as a process of recognition by a non-statutory organisation such as a grouping of 
universities or membership organisation (such as ICDE itself). 

3.3.3 Accreditation-based systems 

Accreditation is interpreted as a process conducted by a national regulatory agency, or similar, 
empowered with ensuring that institutions comply with the set of defined requirements to operate as a 
Higher Education Institution and offer defined qualifications and awards. 

The terms of accreditation may define both the physical and human resources deemed necessary. In its 
simplest terms, an accreditation process might operate as a strictly applied compliance test or as more 
flexible review of an institution’s capabilities of meeting national standards. 

Institutional accreditation against norms may happen within three environments: 

1. Those where the government acts as a regulator, (and often as a funder of a large part of the 
system), providing institutions licenses to operate against well-defined parameters. 

2. Those where there is very little distinction between institutions and the government, with 
government policy significantly implemented within and by institutions. 

3. Management of accreditation processes by an independent agency, but access to governmental 
funding is dependent on accreditation by the agency. 

In unregulated systems, where all Higher Education institutions (or sometimes only certain transnational 
education providers) are operating without formal regulation concerns over quality and consumer 
protection have been powerful drivers for the introduction of mandatory accreditation. 

3.3.4 Norm-based systems 

Accreditation is sometimes operated as a tool to ensure conformity with norms for staffing, 
accommodation and resourcing and may be based on traditional expectations of Higher Education 
institutions. Norms are often a blunt and inflexible instruments. For distance/online education, the 
is to secure adaptation of norms to address relevant factors in such a way as to encompass 


Norm-Based Accreditation 



Process-Based Enhancement 



Complete report. May 2015 


Page 25 of 52 



INTERNATIONAL 
COUNCIL FOR OPEN AND 
DISTANCE EDUCATION 


Quality models in online and open 
education around the globe: State of 
the art and recommendations 



open and distance learning methodologies without undue prescription of institution’s operating model or 
inhibiting future technological innovation. 

There is the risk that attempts to embrace distance education might result in the creation of inappropriate 
and rigid norms that lock distance and online education providers into a specific set of pedagogies and 
technologies. 

High Levels of reporting may be required to demonstrate compliance with standards and norms with 
significant institutional effort consequently invested in maintaining and presenting data at the expense of 
improvement and enhancement activities. 

Norm-based Accreditation systems may be perceived as control mechanisms that restrict innovation and 
change. 

3.3.5 Maintenance of standards 

Accreditation-based systems may reach a state of stability where norms are routinely adhered to as 
indicated by regular reporting. Institutions operate internal data collection and monitoring systems that 
provide assurance of adherence to norms, but do not address quality improvement in a structured fashion. 
The bureaucracies of agencies and institutions conduct dialogues that bypass the academic activities of 
the institution. 

These systems are unlikely to encourage innovation and improvement and inappropriate norms may 
inhibit the launch of open and distance learning initiatives as their operation would require significant 
deviation from existing standards and norms. 

3.3.6 Quality assurance-based 

Institutions operate systems of quality assurance that are based on principles of the quality movement 
requiring higher levels of engagement over and above simple reporting of statistics. These systems 
generally depend on the interrogation of some form of institutional self-assessment structured by 
nationally determined activity areas. Originating in practices in North America, Europe and Australia, these 
principles form the core of most current quality assurance systems. The operating agencies generally 
have a degree of separation from direct governmental involvement and may be representative of or even 
“owned” by the university sector. Peer participation in review processes is generally employed and 
engagement with both staff and students is a routine component of reviews. Review outcomes may 
present both an evaluation of current performance against broadly interpreted national standards and also 
recommendations for improvement. 

They share much in common with the principles of the industrial quality movement in focusing on clarity 
of organisational purpose that is cascaded through the organisation so as to ensure that individuals 
understand the contribution of their role and how improvements may impact on overall achievements. 

Successful implementation of these systems should result in incremental improvement of institutional 
performance and departments/ individuals should be able to identify their contributions to these. When 
these conditions pertain, the systems operate in a fashion consistent with the principles of the quality 
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Though these systems may allow institutions significant scope to define their own mission and purpose 
within the prevailing national context, they may still present significant challenges for open and distance 
learning institutions if defined activity areas are rigidly interpreted by reviewers or institutional managers 
and do not allow for variation in pedagogic methodologies and changing technologies. 

Providing open and distance learning institutions are allowed scope to present and justify their modes of 
operation, there few reasons why they should not operate effectively within this sector of the quality 
assurance spectrum. 


3.3.7 Mature enhancement based systems 

In some instances, national Quality Assurance Systems enable institutions to operate their own internal 
quality assurance processes with light touch oversight by national bodies. National standards are adhered 
to across Higher Education institutions; their students and sponsors have broad confidence in the Higher 
Education system. Internal quality assurance systems are mature and review processes may be used to 
ensure that these systems operate effectively. Consistent with the principles of the quality movement, the 
focus is on improvement and the main function of external review is to ensure institutional focus is on 
enhancement of teaching quality. Systems should be neutral in respect of selection of pedagogy and 
technology and supportive of justified pedagogic innovation. 

This analysis presents a journey to maturity for quality assurance regimes. The starting point will vary 
dependent on the national context, but the direction of travel should be towards increasing reliance on 
institutional processes matched to institutional mission rather than adherence to strict norms. 

3.4 Quality Standard Models 

The analysis on the quality models confirms research by Ossiannilsson (201 2) that most quality standard 
models relate to three to six main dimensions, as for example described in the E-xcellence by EADTU 
(Williams, Kear & Rosewell 2012), and also confirmed by Bacsich (2009-2011). Those three main 
domains are sub-divided into six areas, set out in Figure 1 below: 


Services 

Products 

Management 


• Staff support 

• Student support 

• Curriculum design 

• Course design 

• Course delivery 

• Strategic planning and 
development 
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Fig 1 Three significant main areas related to quality in online learning, including e-learning 

(Ossiannilsson 2012) 

Though some in the educational world may feel uncomfortable with the titling of the areas, they do 
represent a logical presentation of provision of education within a strategically managed organisation. 
Direct mapping to a specific traditional academic organisational structure might not easily translate to the 
structures best fitted to innovative modes of educational delivery. 

In the context of unbundled education, separation into these three areas could be seen as a coarse 
grained model of how the disaggregated service components of in the unbundled world might be grouped 
for quality assurance purposes. 

The methodologies reviewed show a great deal of common ground with differences largely in the grouping 
and detailed granularity of the topic areas and benchmarks. The typical structure of the documentation 
associated with the quality tool is set out below, as in Fig 2, from EADTU E-xcellence, the first part on 
Strategic Management and the first Benchmark no 1 . 


The institution has an e-learning strategy that is widely understood and integrated into the 
overall strategies for institutional development and quality improvement. E-learning 
policies conform to legal and ethical frameworks. 

All excellent Some excellent Adequate Mainly adequate Inadequate 

□ □ □ □ □ 

Please add your comments or refer to evidence: 


Fig 2 The first part on Strategic Management and the first Benchmark no 1, from EADTU E-xcellence 

From this statement cascade further subsidiary statements, possibly four or five, presenting contributory 
components e.g. “1.1 The plan outlines the physical and human resource implications”. Each of these is 
accompanied by performance indicators and a guidance note on sources of evidence to demonstrate 
compliance. Many systems include some form of scoring system, typically based on a Lickert scale that 
can be used for advisory purposes if the system is designed for internal use within institutions, but may 
be numerically scored if the system is implemented by an accreditation agency. 

Systems may be paper-based but several offer online versions with automated generation of performance 
profiles. Though typically structured around 6-10 main topic areas, the extensive use of subtopics can 
result in frameworks with in excess of 100 scored items. Arguably, the more numerous the topics, the 
more constrained the system may be in its application, as it may define levels of activity that are not found 
in all institutions. This is a particular risk in systems that emanate from the practices of a single institution, 
are not generalised and may include very specific performance indicators. 

Implementation of accreditation and certification schemes is dependent on peer review teams 
interrogating documentation and staff to verify the statements and assertions made in documentary 
submissions. The availability of suitably experienced and trained personnel has been seen as a barrier 
to implementation, but increased use of open and distance learning techniques should reduce this 
challenge. 
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Use of the systems to develop an institutional improvement plans is a specific objective of several of the 
schemes reviewed. The EADTU E-xcellence scheme makes this a specific requirement of for the award 
of the E-xcellence Associate in Quality Label with periodic re-assessment focusing on progress in 
implementation of the improvement plan. 

Below in the quality matrix follow some examples from our reviewed quality standard models, and those 
represented here are some of the most used and well-known quality models. In the matrix, they are related 
to the purpose of the model. 

3.5 Quality Matrix 

In the second stage of review, we clustered the Quality Standard Systems from a selection of those 
considered the first round, e.g. those which were in use, covering all (most continents), target groups, 
stakeholders, post online Higher Education (OER; MOOCs and similar Academic Partnerships). Other 
criteria for clustering were global and/or national level, and their intended use in benchmarking, 
accreditation or certification, Table 1. In addition, authority body, e.g. association, agency or just as a 
theoretical or practical frame of reference, were also seen as criteria. 

Table 1 Most Used Quality Models 



ACODE (the Australasian Council 
of Open, Distance and e- 
Learning) 

AVU (the African Virtual 
University) 

CALED (the Latin American and 
Caribbean Institute for Quality in 
Distance Education) 


CHEA (the Council for Higher 
Education Accreditation), US 

E-xellence 

EADTU (the European 
Association of Distance Teaching 
Universities), NL 

OpenupEd 

EADTU (the European 
Association of Distance Teaching 
Universities), NL 
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UNIQUe 

EFQUEL(the European 
Foundation for Quality in e- 
Learning), BE 




ECB Check 

EFQUEL (the European 
Foundation for Quality in e- 
Learning). From Dec 2014 GIZ 
(Deutche Gesellshaft fur 
International Zuzammenarbeit), 
DE 




The eLearning guidelines (eLg) 

Ako Aotearoa, developed by 
Tertiary Education Commission, 
led by AUT University and Massey 
University, New Zealand 




The E-Learning Maturity Model 
(eMM) 

New Zealand Ministry of 
Education Tertiary E-Learning 
Research Fund 




E-learning Quality Model (ELQ) 

NAHE (The Swedish National 
Agency for Higher education) 




Epprobate 

The Learning Agency Network 
(LANETO e V), DE 




Khan eight -dimensional e- 
learning framework 
Badrul Khan 




The OLC Quality Scorecard 

Online Learning Consortium, 
(former Sloan-C), US 




OER TIPS 

The Commonwealth Educational 
Media Centre for Asia (CEMCA) 




Pick&Mix 

Matic Media, SERO Consulting 
Ltd, UK 




3.6 Characteristics of Quality Systems 

Our analysis yielded a set of elements, which we found characterise the majority of the systems analysed, 
and which, in our view should form core characteristics of any quality system for e-learning. As such, they 
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can be seen as both useful for a high-level analysis of the field and further as a framework for improvement 
and benchmarking of the systems themselves. 

Thus, we find that quality systems in the field are: 

• Multifaceted - systems use a multiplicity of measures for quality, and will often consider strategy, 
policy, infrastructure, processes, outputs and more so as to come to a well-rounded view of holistic 
quality. 

• Dynamic - flexibility is built in to systems, to accommodate for rapid-changes in technology, as 
well as social norms. For this reason, they rarely refer to specific technological measures, and 
rather concentrate on the services provided to users through that technology. 

• Mainstreamed - while all the quality tools surveyed aim at high-level quality improvement, this is 
intended to trickle down throughout the institution and be used as a tool for reflective practice by 
individual members of staff in their daily work. 

• Representative - quality systems seek to balance the perspectives and demands of various 
interested stakeholders, including students, staff, enterprise, government and society at large 

• Multifunctional - most systems serve a triple function of instilling a quality culture within an 
institution, providing a roadmap for future improvement, as well as serving as a label of quality for 
outside perspectives 

Furthermore, the majority of systems reviewed are designed to be complementary with the dominant, 
institutional self-assessment based, model of quality assurance thus integration of quality assurance with 
that of its conventional provision is in most cases a practicable option. Whatever quality model, which is 
chosen or used, existing ones, or ones developed in the future, the above mentioned characteristics have 
to be considered. Above that, considerations have to cover a set of principles, such as: 

• Contestable/ debatable - As there are many stakeholders with a variety of interests, as well as 
quality is multifaceted, etc., there are many opinions that constitute bad, good, and or excellence 
quality. While the review systems are designed to provide a structure for objective assessment of 
quality, it is difficult to remove all elements of subjective judgement, as quality used to be 
described as being ... “in the eye of the beholder”. Those systems that invoke peer review and 
seek to form a community of users assist in the development of shared perceptions of quality 
levels. 

• Context bound contextualises generic vs content, e.g. subject-based - There is a tension 
between whether quality can be based on generic dimensions, and or if quality is content-/subject- 
based. In the context of open and distance learning, the delivery and support mechanisms must 
have parity with academic rigor if effective teaching is to be delivered. Hence, local contexts of 
culture, language and infrastructure will influence assessments of institutional quality, 

• Open culture/practice core of culture - The way we measure and look at quality will surely differ 
in emerging open cultures and practices in changing learning landscapes and unbundling contexts 
and with increased personalisation. Flexible systems for quality assurance offer better prospects 
for adaptation to changing practice and effective operation of improvement strategies will facilitate 
innovation. 

• Personalisation - Personalisation of learning and education is more and more valued, and there 
rv ' i qht be tensions what that means for quality and quality dimensions, as those are often set from 

anisation’s points of view. As Bates (12/01/2015) phrases it: ...we will not talk about online 
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learning in the near future (2020). The future is about choices and this gives consequences for 
students and learners, for faculties and instructors, for institutions, as well as for governments. 
Accessibility, related to individuals with special needs, is strongly related to personalisation, and 
this feature needs to be addressed in any quality model, if not it is a gap in quality enhancement 
and quality assurance. 

Those set of characteristics might have impact on how quality in e-learning, online learning is discussed. 


3.7 Nature of Quality Interventions 

We find that the type of quality system applied, and/or the way in which a particular system is applied, 
varies widely depending on the institution’s maturity with e-learning processes, Table 2. Based on the 
stage at which an institution is, we find differences in: 

• the type of quality criteria and methods employed 

• the role of quality managers, reviewers and external assessors 

Table 2 Quality interventions 



Initial / Early 
Stage 

Developing 

Mature 

Evolving 

Stage 

description 

planning or just 
beginning to 
introduce e- 
learning 

e-learning is 
introduced, but 
system is scaling 
up, and processes 
are still in flux 

the e-learning 
system is 
established, and 
running using well- 
established 
processes 

the institution is 
moving beyond 
well-established 
processes to 
achieve excellence 

Purpose of 

quality 

schemes 

provide a template 
for system-design 

verify correct 
implementation of 
processes - 
address failures 

stimulate and 
support continuous 
improvement 

provide an 
excellence 
benchmark 

Role of 
quality 
managers 
/reviewers 

consulting as to 
implementation of 
the standard 

auditing as to 
whether the 
specificities of the 
standard are 
reached 

monitoring of 
processes and 
outputs, and 
recommending 
improvements 

benchmarking 
outputs against 
excellence criteria 


3.8 Stakeholder Perspectives 

The major stakeholders in the processes of education are the institutions, their staff and students, 
governments and other funders, professional bodies, employers, and the general public as direct or 
indirect consumers of the outputs of the educational system. Their interests will variously focus on the 
quality of educational experience provided to students, the contribution graduates make to the economy 
and national well-being and the costs and efficiency of the system. Protection of the interests of these 
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stakeholders is among the reasons for the introduction of national quality assurance regimes. There may 
be particular interests in the application of regulation and quality assurance to novel modes of educational 
delivery to protect existing institutions from international competition, but analysis at the macro level of 
the political complexities lies outside the scope of this study. The analysis presented focuses at the meso 
level and concerns stakeholder groups identifiable at institutional level. 

3.8.1 Perspectives of institutional stakeholder groups 

Many of the methodologies reviewed make reference to addressing the needs of stakeholder groups 
within their criteria particularly in those sections addressing institutional strategy and selection of 
curriculum. However, the Ako Aotearoa-funded project led by New Zealand Tertiary College (2014 is 
unique in developing e-learning guidelines for quality in e-learning, that consider both stakeholder 
perspectives and aspects of institutional maturity aspects. The e-Learning guidelines (eLg) have been 
developed to assist the tertiary sector in its engagement with e-learning. The guidelines offer prompts for 
reflection from five perspectives - the learner, teacher, manager, organisational leader and quality 
assurance body. When considering one of these perspectives in the e-learning and e-teaching process, 
the guidelines assist the design, implementation and enhancement of practice to ensure thoughtful and 
effective e-learning provision. Table 3 provides an exemplar of the document’s structure and approach. 
The stakeholder perspectives are maintained in the more detailed topic by topic guidance. 


Table 3 Stakeholders perspectives and maturity levels (eLg 2014) 



Designing 

Implementing 

Enhancing 

Learners 

perspective 

The planning, design and 
preparation of e-learning and 
assessment materials for 
delivery to a set of learners. It 
considers collaborative 
design, teaching, learner 
skills, needs and support (p 
4) 

To support the effective 
delivery of e-teaching and e- 
learning. It considers 
teaching, collaboration, 
professional development 
and technical aspects (pp 4- 
5) 

The forward planning for 
ongoing improvement and 
sustainability of e-learning. It 
considers ongoing 
professional development and 
the gathering and use of 
evidence for continued 
improvement and 
effectiveness (p 5) 

Teacher 

perspective 

The planning, design and 
preparation of e-learning and 
assessment materials for 
delivery to a set of learners. It 
considers collaborative 
design, teaching, learner 
skills, needs and support (p 
9) 

To support the effective 
delivery of e-teaching and e- 
learning. It considers 
teaching, collaboration, 
professional development 
and technical aspects (pp 9- 
10) 

The forward planning for 
ongoing improvement and 
sustainability of e-learning. It 
considers ongoing 
professional development and 
the gathering and use of 
evidence for continued 
improvement and 
effectiveness (p 1 0) 

Manager 

perspective 

The planning, design and 
preparation of e-learning and 
assessment materials for 
delivery to a set of learners. It 
considers collaborative 
design, teaching, learner 

To support the effective 
delivery of e-teaching and e- 
learning. It considers 
teaching, collaboration, 
professional development 
and technical aspects (p 17) 

The forward planning for 
ongoing improvement and 
sustainability of e-learning. It 
considers ongoing 
professional development and 
the gathering and use of 
evidence for continued 
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Organisation 
al leader 
perspective 


Quality 

assurance 

perspective 


skills, needs and support (p 
17) 


improvement and 
effectiveness (p 18) 

The planning, design and 
preparation of e-learning and 
assessment materials for 
delivery to a set of learners. It 
considers collaborative 
design, teaching, learner 
skills, needs and support (p 
23) 

To support the effective 
delivery of e-teaching and e- 
learning. It considers 
teaching, collaboration, 
professional development 
and technical aspects (p 23) 

The forward planning for 
ongoing improvement and 
sustainability of e-learning. It 
considers ongoing 
professional development, and 
the gathering and use of 
evidence for continued 
improvement and 
effectiveness (p 24) 

The planning, design and 
preparation of e-learning and 
assessment materials for 
delivery to a set of learners. It 
considers collaborative 
design, teaching, learner 
skills, needs and support (p 
28) 

To support the effective 
delivery of e-teaching and e- 
learning. It considers 
teaching, collaboration, 
professional development 
and technical aspects (pp 
28-29) 

The forward planning for 
ongoing improvement and 
sustainability of e-learning. It 
considers ongoing 
professional development and 
the gathering and use of 
evidence for continued 
improvement and 
effectiveness ( p 29) 


3.8.2 Work of specific international stakeholders 

UNESCO and OECD together published the Trans National Guidelines for Quality Provision in Trans 
National Higher Education to assist in understanding the issues associated with liberalisation of cross 
border, or transnational education. While the guidelines do not make specific reference to online or post- 
traditional Higher Education, they create a set of overarching recommendations for governments in 
dealing with quality of cross border education, including an invitation to: 

• Governments to establish comprehensive systems of quality assurance and accreditation for cross 
border Higher Education, recognising that this involves both sending and receiving countries. 

• Higher Education institutions and providers to ensure that the programmes that they deliver across 
borders and in their home countries are of comparable quality and that they also take into account 
the cultural and linguistic sensitivities of the receiving country. 

• Student bodies to get involved as active partners at international, national and institutional levels 
in the development, monitoring and maintenance of the quality provision of cross border Higher 
Education. 

In particular, the recommendations on ensuring “comprehensive systems” of quality assurance puts the 
onus on governments to revise current systems to take account of developments being realised to opening 
up of education and unbundling as discussed earlier. 

In the UNESCO 2012 Paris declaration, it is recommended that member states “promote quality 
assurance and peer review of OER. Encourage the development of mechanisms for the assessment and 
certification of learning outcomes achieved through OER.” In its communication on “Opening up 
Education: Innovative teaching and learning for all through new Technologies and Open Educational 
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Resources” the European Commission recommends the creation of “new quality frameworks for OER and 
mapping with curricula”. 

The US Council for Higher Education Accreditation (CHEA) has established a specific group of 
international experts to specifically task with discussing issues surrounding OER, open education, post- 
traditional education and other challenges facing education, and providing policy advice on how to deal 
with them from a quality standpoint. 

The International Standards Organisation (ISO) has set up a Project Committee to propose a Quality 
Management Standard for Educational Organisations (ISO 21001) - providers of e-learning have been 
included as a specific target of the standard. 

The European Association of Distance Teaching Universities created the OpenupEd Consortium - an 
alliance of MOOC providers who all agree to follow the same quality principles and practices. 

3.8.3 National approaches to quality 

At national level, governments and quality assurance agencies often work closely on issues surrounding 
recognition, accreditation and quality assurance. With respect to quality assurance and accreditation in 
particular, four approaches have been identified in dealing with the challenges posed by e-learning and 
distance education: 

• Creation of specific criteria: several countries have specific, comprehensive sets of criteria for 
e-learning providers, and/or distance teaching institutions. 

• Mainstreaming into overall quality assurance: several other countries, have updated or 
reviewed their existing quality assurance criteria, and found that a single set of criteria can cover 
all types of institutions. A notable example of this is the UK, which moved from advisory guidance 
in its code of practice to a mainstreamed system that is neutral on modes of delivery. 

• Hybrid/personalised system: while at the moment only partially implemented or under 
discussion, quality assurance systems can have a standard ‘core’ applicable to all kinds of 
education and organisations, with add-on ‘modules’ specific to distance or e-provision. 

• No approach - other systems have not considered the impact of e-learning onto their criteria, 
creating sometimes perverse results, such as limitations on the size of classrooms, or 
requirements for physical facilities which are not required for e-learning. 

3.8.4 International coordination of quality systems 

Trans National Education, and in particular e-learning has been a frequent subject of trade negotiations 
in recent years, in particular of the: 

• General Agreement on T rade Services (Multilateral - World T rade Organisation) 

• Directive on Services in the Internal Market (Multilateral - European Union) 

~ nsatlantic Trade and Investment Partnership (Bilateral - EU-US) 
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The aim of such negotiations is a harmonisation of regulations around trade, to remove unnecessary 
barriers created by over-regulation. Examples of such barriers include : 9 

• Restriction on electronic transmission of course material 

• Non-recognition of degrees obtained through distance mode 

• Prescription of minimum standards or attainments 

• Insistence that providers be accredited in the home country 

• Insistence on a local partner to allow provision 

While in theory, the removal of such barriers would aid internationalisation of education, and digital 
education in particular, the inclusion of education in these trade negotiations is widely opposed by 
university groups, teacher unions, student unions and others, on the grounds that national autonomy over 
quality standards might be lost, with a consequent weakening of national quality standards. 

However it must be recognised that in some contexts there are strong market motivations for individuals 
to secure transnational qualifications, whether formally recognised or not, in the belief that they will provide 
an advantage in securing private and multinational sector employment. The MOOCs movement is also 
promoting the value of additionally in the range of formal and informal credentials that individuals might 
hold citing the advantages of access to the teaching of world class institutions to students in nations whose 
educational systems are challenged through lack of resources. 

Resolution of the stakeholder tensions between protection of national systems and access to high-quality 
services irrespective of origin is currently best handled in those countries where transnational education 
is regarded as a valued national business. Regulation and quality assurance requirements acknowledge 
the importance of institutional legitimacy and adherence to quality assurance standards of both host and 
originating countries in some instances invoking joint review activity between agencies. Instances of good 
practice are evolving in South East Asia and the Middle East. 

In an increasingly complex and changing environment for transnational education provision all legitimate 
stakeholders have a responsibility to work collaboratively to protect students from degree mills and bogus 
accreditation and credential systems. 


idoc.unesco.orq/imaqes/001 4/001 473/1 47363e.pdf 
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4 Discussion and Conclusions 

Internationally, there is a movement towards convergence in the processes of quality assurance in Higher 
Education through the work of regional and global bodies, such as ENQA, CHEA, APQNE, INQAAHE, 
etc. Universities are complex and sophisticated institutions; their “products” are not tangible and readily 
assessed against definable physical norms hence any process of evaluation is likely to have a significant 
subjective and comparative component. The emerging standard model is one based on the preparation 
and review of institutional self-assessments that give institutions scope to describe their institutional 
missions and the processes in place to deliver them. In the Higher Education sector peer, review forms 
an important component of many quality assurance systems with peer judgements standing in place of 
defined standards and norms. 

In differing national and regional contexts, it is possible to identify a spectrum of quality systems ranging 
from tightly defined regulatory systems to more open systems allowing institutions significant latitude in 
the interpretation of national standards. Models of the progression to maturity of quality assurance 
systems have been developed and presented, which address approaches to quality at macro level 
(national systems), meso level (institutional) and micro level (team or individual practice) Table 4. 

Table 4 Quality Assurance Agencies vs Institutions 


QA AGENCY ► 

Institution 

▼ 


Institutional Review 
based 


Enhancement 

Focused 




Institutional 
Review Based 




Norm Based 





In parallel, it is possible to identify institutional journeys to maturity in their use of e-learning. In many 
conventional institutions, e-learning activities had their origins in initiatives at departmental or even 
individual level. With this cottage industry/bottom up style of development expansion was often slow with 
little experience sharing and knowledge transfer between departments. While students may enjoy 
stimulating teaching, quality assurance and resilience is highly dependent on individual rather than 
institutional systems. The introduction of Virtual Learning Environments (VLEs) as essential components 
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of institutional infrastructure enabling e-learning and off campus provision has ensured that the majority 
of institutional strategies acknowledges the role of e-learning and increasingly addresses its development 
in institutional strategy. In the US, the Online learning Consortium (formerly Sloan Consortium) has 
tracked these trends in its annual surveys 10 with the most recent data indicating online provision is 
regarded as critical to future strategy in 70% of institutions. In the cascade of institution policy downwards 
to departmental level top-down and bottom up innovation meet. Negotiation of appropriate internal quality 
assurance processes that both ensure conformity to institutional norms and encourage innovation can 
present a challenge. 

Applying these concepts of progression to maturity in quality assurance to both institutions and national 
systems offers the possibility to classify the prevailing context for particular institutions and identify routes 
for improvement. 

At the lowest levels of maturity (cell 1.1), quality assurance systems are norm-based and institutional 
systems confined to reporting of compliance. Poorly resourced quality assurance agencies may face 
challenges associated with regulation of rapidly expanding private providers whose motivations range 
from the high levels of philanthropy to profit maximisation. Regulation of online learning adds significantly 
to the difficulties these organisations face and progression to higher levels of maturity may be challenging. 

We speculate that in the majority of European, North and South American and Asian countries cell 2:2 
best represents the situation pertaining with stable quality assurance regimes and many institutions 
aspiring to higher levels of performance. However, the challenge of integrating non-traditional teaching 
modes may not be fully resolved in all cases 

At the highest level of maturity (cell 3.3), institutions meet their institutional objectives with processes in 
place to ensure enhancement. At this level, quality assurance systems have become neutral on mode 
and technologies of delivery enabling institutions to establish their own mix of campus and off campus 
delivery modes to meet their institutional objectives. 11 

Distance education and online providers with their reliance on systematised approaches to curriculum 
development are likely to outperform their conventional counterparts in quality assurance regimes 
operating at levels 1 and 2 in which clarity of institutional purpose and availability of data predominate. 
However, the rigidity of their systems may prejudice their abilities to show flexibility in timely enhancement 
of teaching. 

This matrix is presented as a simplistic model, but offers scope for refinement through examination at 
macro, meso and micro levels. 


10 http://en.wikipedia.org/wiki/Benchmarkinq e-learninq#eMM ,28e-learninq Maturity Model. 29 

11 The UK’s Quality Assurance Agency, as an exemplar, earlier included specific references to distance and e- 
learning in sections of its code of practice relating to Collaborative provision. Following consultations in 2010/11 its 
codes have been updated and now contain no specific conditions relating to these methods of delivery. Their 
expectation is that ail institutions will deploy technology-enhanced learning in some form in their provision but they 
leave determination of styles, services and quality assurance as institutional responsibilities. The UK Quality 
Assurance Agency stance may be the most liberal but other western developed world agencies adopt similar 
approaches supplemented by advisory documents or referencing such documents developed by other agencies. 
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While the detailed terminology and grouping of criteria might differ, all quality models we reviewed 
addressed the major areas of strategic management, curriculum and course development, and the 
support of students and staff. The specifics of indicators and benchmarks showed differentiation in 
emphasis and style, but none stood out as adopting a unique approach. Some presented quantified 
performance indicators in areas, such as staff and student ratios addressing both teaching and 
administrative functions. Other systems concentrated on principles rather than tight specification. 

The documents reviewed fell into three broad classes: 

• Accreditation and recognition requirements issued by regulatory bodies 

• Labelling and Badging methodologies and schemes operated by membership organisations 

• Advisory documents and resources from a variety of sources 

The majority of the systems reviewed emphasised the need for a holistic approach to quality and in the 
use of the quality dimensions adopted, however, some of the accreditation systems placed significant 
emphasis on baseline staffing and resource requirements. 

In undertaking a study such as this, the influence of history in the development of standards and norms 
becomes apparent through the legacy of successful innovations shaping the expectations of subsequent 
developments. The accreditation models examined can be aligned with the developments mapped by 
Taylors Five Generations of distance learning that characterised the media centred approaches to 
distance and online learning of the large scale institutions of 1970s-1990s. 12 

Institutional case studies presented in Jung, Wong, Belawati illustrate the adoption by Open University 
Hong Kong and UNISIM of Open University UK-derived quality assurance processes, negotiation of their 
recognition with local accreditation agencies and their subsequent modification to meet the differing 
requirements of institutions operating at reduced scale and with more limited resources. Similarly the COL 
supported DEMP project embedded and codified the quality assurance experience of Open University Sri 
Lanka. Through the work of ACDE, the DEMP model has been modified and is now being promoted for 
application in African distance education systems for institutional review purposes. Staffing norms derived 
from IGNOU experience are embedded in India’s Distance Education Council (DEC) accreditation criteria, 
but the future direction of quality assurance of distance learning in India is uncertain since the transfer of 
responsibility of DEC from IGNOU to Ministry Education. 

Use of ISO 9000/1 systems as the scaffolding for the creation of internal quality assurance systems is 
illustrated in the development of Universitas Terbuka and Korean Cyber University systems demonstrating 
that standard quality systems approaches are applicable in developing systems for distance and online 
learning. However, this pattern of pioneer institutions influencing the development of national systems 
there is an inevitable lag between innovation and its mainstream into formal regulatory systems. 


12 Developed by Professor Jim Taylor in late 90s and early 2000s as a sequential development model, based on 
introduction of new technologies into distance education. As many Quality Assurance agencies have yet to engage 
ly generations of the Taylor model it illustrates of how regulators can lag behind the innovation. 

;3I. uni-oldenburq.de/cde/media/readinqs/taylor01 .pdf 
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Many of the quality models reviewed gained prominence in the early years of the century and reflect the 
processes associated with content rich forms of online delivery corresponding with the final tier of the 
Taylor model. Thus, the experience of the industrialised print-based approach of the early generations of 
Taylor’s evolutionary model is captured and lives on in some accreditation systems, whereas more recent 
developments have yet to be generalised and standardised. 

There is little evidence of the experience of online-community focused education systems being 
embedded in accreditation systems and any identified national accreditation system that explicitly 
embraces their use within its criteria have not been found. 

In several countries, there is evidence of Quality Assurance Agencies seeking to add guidance and 
informal standards relating to distance and online education to their existing systems of institutional quality 
assurance. Notable examples are the Malaysian Qualifications Agency, which provides advice for 
institutions already accredited, but seeking to extend their activities and Nigeria’s National University 
Commission (NUCN) that has a supplementary accreditation process designed to support expansion of 
distance learning provision. Neither of these adopts an overly prescriptive approach. In Malaysia, it 
appears that distance and online learning will be reviewed as an integral component of the institution’s 
provision in line with the increasing maturity of a quality assurance regime that has progressed from 
control and regulation of the large private Higher Education sector to oversight of quality in both public 
and private sector institutions. In Nigeria, quality assurance processes are at a lower level of maturity and 
the NUCN aim is to both improve quality in conventional teaching institutions and enable them to increase 
capacity significantly through expansion of distance education provision. Their distance education 
guidelines have both a facilitative and regulatory role providing guiding principles for the development of 
new modes of delivery and a non-prescriptive approach to consideration of institutional proposals. 
Proposals are scored in the review process, but no absolute norms are applied. It is believed that this 
approach will have a positive impact on distance education and conventional sectors to the benefit of all 
stakeholders. In Kenya, the Commission for University Education guidelines devotes a chapter to distance 
education offering a flexible, well-balanced presentation of requirements. 

Whether addressed at macro, meso- and micro level. There is not one quality model which will fit for all 
contexts. It is obvious that there are needs for cultural and contextual adaptations, but there is a set of 
principles for quality in online learning, which allow flexibility and dynamic changes to embrace new 
technology and pedagogy. These principles have been presented and discussed. 

The growth of the MOOCs movement again raises issues relating to the function and practice of quality 
assurance. The separation of services lends itself to independent quality assurance of each component 
to appropriate industry norms: media development, customer service provision, etc. However, quality 
assurance of the full package experienced by students remains unresolved. The traditional universities 
leading the movement have established their reputations on the basis of their campus-based teaching 
and, in many cases, the close association of quality with highly selective intake and excellence, a notion 
that runs counter to the principles of openness espoused by some sectors of the distance teaching 
community. The stance of Quality Assurance Agencies is cautious, they have no mechanisms in place for 
MOOCs; however, encourage institutions to adapt the principles and processes used in general provisions 
to MOOC activity in order to preserves institutional and national reputation. However, the impact of 
MOOCs remains extremely limited in comparison to the overall scale of global Higher Education provision 
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and institutions at the core of the open education movement, such as EADTU, are active in pursuing 
principles based and student focused approaches to the quality assurance of MOOCS. 

This study showed well that e-learning today needs to be mainstreamed as we globally are living in an e- 
society, and in many regions digital native learners are in majority nowadays at Universities around the 
globe. There are also worldwide debate on opening up education and use the potentials of increased 
digitization. Likewise, it has been obvious that there are needs to include post online Higher Education, 
like OERs and MOOCs, and informal learning into quality related issues. Thus, there are questions if 
quality assurance bodies have the capacity to extend their work beyond traditional Higher Education 
providers? 

The MOOCs movement has emphasised innovation and mass participation outside the mainstream of 
Higher Education, free mass participation placing it outside the sphere of influence of quality assurance 
regimes intended to ensure quality provision for registered students supported through public or private 
purse. The commercial MOOC platforms will focus their attention on the quality assurance of production 
and support service provision. Models of MOOC production for the content focused xMOOCs may share 
common features with the modes of module production employed in many distance education institutions 
being reliant on close collaboration between academic subject matter experts, instructional designers and 
media professionals; hence, quality assurance processes may be interchangeable. Similarly principles of 
student support quality assurance may also be transferable, though the larger populations may demand 
differing service standards than those prevailing in the distance education sector. There is already 
evidence of initiatives to adapt and transfer methodologies between the standard system and MOOC 
space. For example, EADTU’s OpenupEd project proposes quality assurance criteria for MOOCs that are 
derived from criteria and benchmarks derived from its earlier E-xcellence scheme. There are indications 
that statutory Quality Assurance Agencies are exercising caution regarding formal accreditation of 
MOOCs that lie outside their current remits. For example, the UK Quality Assurance Agency has recently 
published a statement on MOOCs . 13 It outlines their interests in protection of student interests and of the 
reputation of UK Higher Education noting that while it has no formal responsibility for the quality 
assurance, it is actively reviewing developments and those institutions may wish to refer to their MOOC 
activity in self-assessment and audit documents they prepare for Quality Assurance Agencies reviews. 

Thus, in this innovative phase, users have two broad indicators of quality, the reputation of the platform 
provider and the reputation of the Institution based on its performance and standing of its mainstream 
teaching activities. 

MOOCs are a high-profile example of more general trends for unbundling of education in which learners 
may use multiple sources to assemble learning packages suited to the specific requirements. They might 
access content from one source, tutorial support from another and assessment/certification from another. 
In this unbundled world, each component might be quality assured by different organisations 
implementing standards relevant to the industry sector, training material development, customer service 
etc. 


13 http://www.qaa.ac.uk/en/Publications/Documents/QAA-position-statement-MOOCs.pdf 
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As delivery of teaching and support services via mobile devices assumes greater importance both in the 
“full service” institutions and the unbundled education space so engagement with telecoms providers will 
be of increasing importance. They routinely manage customer information, charge for connection and 
data services and in a growing number of countries manage secure financial transfers. Already they 
provide educational services to their customers such as online quizzes associated with school level 
qualifications, language studies . 14 Increased engagement between organisations shaping the future of 
quality assurance and telecoms organisations is of significant importance. 

Agile approaches to quality are recommended as the educational landscape is changing. We are shooting 
at a moving target group. It is obvious that here are needs for both innovation and excellence on quality 
in e-learning and online learning, including post online Higher Education. 

Judging by our survey, our analysis and our recommendations, we can summarise that: 

• There is an extremely large variety of quality tools catering to every variety of audience and need. 

• We have not found a gap in terms of analysis of institutional systems, which would require a new 
scheme to be developed. Although in the case of recognition and unbundling, which are not e- 
learning specific, there are definite deficiencies. 

• We have however found that all the quality systems suffer certain deficiencies (lack of universal 
applicability, unclear which maturity levels they are best for, widely divergent quality of reviews 
and of advice given, challenges to respond to change, etc.). 

• We do not think that an organisation such as ICDE with its limited resources and staff, is in a 
position to address these problems by creating a better quality review service. 

• We do see a role for an international organisation in the following main areas, all of which are 
critical: 

o providing a register of good quality systems, and a guide to members on which ones to 
use for which purpose 

o addressing common issues around training, best practice sharing, localisation, etc., for 
providers of quality systems 

o working with international organisations to ensure a harmonised regulatory environment 
o working with international agencies to ensure student engagement in determining quality 
standards 


Although we tried to cover all continents, through our extended RAG, and in contact with the ICDE SCOP 
presidency, many reports, etc., of the most common and used models, are from western countries and in 
English. Thus, we are seriously aware that there might be gaps in the study and with our results. However, 
from contacts and through desk analyses of reports, we found similarities how quality in online teaching 
and learning are discussed, reflected and measured with progression towards a unified approach. 


From the result of the study, it is recommended that ICDE and COL might work together with the mobile 
telecom sector, together with educational experts, to create a set of quality criteria specifically related to 
mobile learning systems delivered by feature-phone/low-end (cheap inexpensive smart phone, basic 
smart phone functions) smart phones. Given the lag between technical innovation and implementation of 
quality standards, this is one area with potentially very high impact that would benefit from concerted 


http://www.airtel.in/education 
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development of appropriate standards that integrate the telecoms sector experience of customer account 
and service management with the requirements of the educational world. 

5 Recommendations 

This research study aimed to result in a set of recommendations and proposals to ICDE for communication 
with their stakeholders, and to set strategic goals. In this section, we present a set of 1 1 recommendations, 
and aligned with the recommendations, we also make some proposals for ICDE for their future work, and 
to be communicated with members, as well as with stakeholders (Appendix 8). 

Recommendation 1: Mainstream e-learning quality into traditional institutional quality assurance 

Our investigations with stakeholders highlighted two related problems. The first of these is an exhaustion 
with existing quality assurance processes, indicating that they are already too resource intensive when 
evaluated on a cost-benefit basis. The second is concern of overlap, or in some cases even contradictions, 
between different quality labels / systems. 

Mainstreaming of ‘e-learning’ into quality assurance requires disaggregation of the common components 
of e-learning quality systems and integrating each of these into the appropriate part of the quality 
assurance process. These are: 

• E-learning as technology-enhanced learning: TEL here mainly refers to technology-supported 
optimisation of learning and teaching processes, but also of governance/administration processes. 
This could include addition of criteria on providing access to high-quality learning resources 
electronically, having appropriate technological infrastructure to support classroom-based 
teaching and/or optimising administrative processes through a paperless workflow to existing 
quality assurance criteria. 

• E-learning as a mode of provision: distance learning, blended learning, problem-based learning, 
lectures, work-based learning and simulation are all distinct forms of provision, which require 
distinct measures of quality. Rather than present an entire quality label for each, a mainstreamed 
quality system would set out a common core of criteria applicable to all teaching and learning, and 
augment these with modules for each mode of learning - with each module containing only those 
criteria which are specific to the mode in question. 

• E-learning as a driver for innovation: most of the e-learning quality assurance systems reviewed 
include a focus on innovation. This is displayed through requirements for innovation strategies, 
rapid iterative review, connection between research and pedagogy and/or an emphasis on 
learning design (which requires knowledge of latest innovations to select the most appropriate 
means to reach learning objectives). Mainstream quality assurance systems tend to focus on 
guaranteeing a set level of quality rather than on acting as a tool for pushing innovation, and could 
thus benefit from the inclusion of such criteria. 

Recommendation 2: Support the contextualisation of quality systems 

Many of the quality systems we studied make socio-economic-cultural assumptions that are not equally 
true in all contexts. Examples of such assumptions include: 
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• Access to high-bandwidth Internet: a stress is often put on interactivity without considering the 
access implications of such interactivity in all places. 

• Existence of a multi-stakeholder, participatory governance environment: a strong feature of 
western democracies, this does not necessarily apply in the case of top-down, less participative 
governance. 

• Personal Computers (PC) are the main way to interact with e-learning, when in fact mobile devices 
are the connection method of choice in many countries. 

• Academia universally speaks English or a national language: most quality assurance models are 
available only in English or in the language of the country from where they originate. 

It would be mistaken to limit ourselves to a context assuming that high quality e-learning can only be 
delivered over high-speed Internet to a Personal Computer, in an English speaking western democracy. 

This would suggest a need for stakeholders to: 

• Translate existing quality systems to allow for wider access (along with providing for multi-lingual 
reviewers with knowledge of the cultures, etc.). 

• Work with quality assurance model providers to widen the interpretation of specific criteria within 
existing systems, to allow for more international applicability, and regional sensitivity. 

• Support the development of regional quality labels, derived from existing tools and adapted for 
context. 

• Explore possibilities of ICDE and COL working together with the mobile telecom sector and 
educational experts to create a set of quality criteria specifically related to mobile learning systems 
delivered by feature-phone/low-end smart phones. This is an area in which there is little current 
experience and could yield major benefits. 

Recommendation 3: Support professional development, in particular through documentation of 
best practice and exchange of information 

All of the systems reviewed took an approach to quality, which required significant knowledge on the part 
of the reviewers, as in many cases the criteria on their own did not specify the best practice being required. 
Criteria were littered with terms such as ‘appropriate, suitable, relevant, standard-practice’ and so on, 
without giving examples of what would be considered to match any of these terms. We see this as a 
substantial risk inherent to many of the systems, as the quality of the reviews conducted is directly 
correlated to and heavily dependent on the knowledge and experience of the reviewers. A secondary risk, 
is that where the standards leave scope for value and/or experience based judgements, there can be 
significant difference between the opinions of different reviewers, depending on their background. 

Practically all review schemes offer some sort of training for reviewers, but follow-up on this is usually 
limited. For this reason, it is critical to support professional development of reviewers by building up 
databases of reference materials, which they can use to better understand their role, and improve their 
skills. 
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From an institutional perspective, Tony Bates points out 15 that “it is one thing to have a set of standards 
for e-learning; it's quite another to implement them. Even rarer are studies that attempt to measure the 
impact of a quality assurance process on actual quality of teaching and learning”. 

In particular, we believe that the need for better documentation and reference materials could be achieved 
by stakeholders through two initiatives, aimed at sharing best practice: 

• Creation of an e-learning quality resource hub - an online collection of research papers, quality 
tools, training materials, etc., which could be useful to both institutions seeking to improve their 
quality systems, and to quality assurance reviewers. We would recommend that such a hub would 
be carefully curated to ensure that it focuses exclusively on the best research and tools, without 
creating unnecessary ‘noise’. Examples of more general hubs in related areas include the OER 
Research Portal , 16 and the Open Education Europa Portal . 17 Also, the EADTU’s EMPOWERING 
Universities initiative guides universities in considering all aspects related to online and open 
education . 18 

• Create a best practice database, containing curated examples of best practices in institutions 
around the world, contributed by quality reviewers from examples in real reviews. The database 
would not necessarily be mapped to any particular quality label/scheme, but rather contain best- 
in-class examples under a number of categories, such as “institutional policies, Human Resource 
development, media design, learning design, learning environment”, etc. The database would be 
of particular use to reviewers in writing recommendations for improvement (since it would provide 
examples of what those recommendations could achieve), and for institutions trying to benchmark 
themselves against others, in preparation for a review, or in the process of implementing 
recommendations. It could be complemented by social functionality in the form of forums, wiki or 
other similar tools. 

• Compile and maintain a register of professional development programmes and training materials 
appropriate for use by institutions and Quality Assurance Agencies. 


Recommendation 4: Communicate and promote general principles 

We believe a holistic and conceptual approach to quality management is essential towards the 
implementation of well-functioning quality infrastructure. To this end, in section three, this study set forth 
a set of universally applicable characteristics for e-learning quality systems, derived from a survey of the 
principles as included within existing quality systems, from stakeholder opinions/positions, and policy 
documents. We find that each of the characteristics is essential and intrinsic to a well-operating quality 
system, whether at programme, institutional or regional/national/international levels. 


15 in his blogpost (15/08/2010) http://www.tonybates.ca/2010/08/15/e-learninq-qualitv-assurance-standards- 
orqanizations-and-research/ 

16 http://www.oerresearchhub.org 

17 http://openeducationeuropa.eu/ 

18 http://empower.eadtu.eu/ 
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We believe that stakeholders should take a primary role in communicating and promoting these general 
quality principles, through a combination of actions: 

• Train reviewers on the quality principles, and providing information about the principles 

• Promote the quality principles to distance education institutions, as a foundation on which to build 
their quality policies, and write their quality manuals. 

• Stakeholders should work towards international adoption and mainstreaming of the principles. In 
particular, we recommend that ICDE engage with ISO, and propose these principles for 
incorporation in the upcoming ISO 21001 standard on Quality Management Systems for 
Educational Organisations. 

Recommendation 5: Assist institutions in designing a personalised quality management system 

This study shows that a wide variety of quality tools are available for e-learning - including tools for 
evaluation, quality assurance, quality enhancement, self-evaluation, benchmarking, certification, 
assessment, standardisation and more. All the tools stress that they should be used in the context of 
creating a quality culture within their institutions. 

On the other hand, in particular with regards to e-learning policy and practice, institutions show a wide 
level of maturity levels, with some needing consultancy on how to develop a quality system for an initial 
foray into the field, and others seek incremental improvements to a well-developed and widely-deployed 
system. 

To this end, several of the quality systems reviewed provide eligibility checks (e.g. UNIQUe), quick checks 
with self-evaluation (e.g. E-xcellence) or other similar tools to allow institutions to determine their eligibility 
to use the tool in question. However, quality-service managers from several of the schemes still mention 
that queries from institutions that are not appropriate or not ready for the schemes remains a recurring 
problem. 


Recommendation 6: Address unbundling and the emergence of non-traditional educational 
providers 

Technological disruption of educational provision is rapidly leading to a situation where any institution with 
sufficient expertise can design and deploy a course globally at minimal cost. Coupled with the 
phenomenon of unbundling facilitated through regulatory changes, pedagogical innovations, and 
technological evolution, a multitude of specialised providers is entering the certification and assessment 
space. While still in their infancy, these changes promise to shake up the entire trust infrastructure that 
has been built around education, particularly Higher Education, in the past years. 

Unbundling means that quality systems can no longer be focused exclusively on educational institutions 
as all-in-one learning design, teaching, testing, and certification providers. New providers are specialising 
in specific functions of educational provision - however, quality assurance standards and other regulatory 
instruments which apply to educational institutions do not necessarily apply to them directly, as they not 
. x D re g U | a t e suc h entities. On the other hand, limitations in legislation could also hinder the same 



Complete report. May 2015 


Page 46 of 52 


INTERNATIONAL 
COUNCIL FOR OPEN AND 
DISTANCE EDUCATION 


Quality models in online and open 
education around the globe: State of 
the art and recommendations 


companies from offering services - e.g. currently only universities can offer ECTS, even though other 
institutions are capable of offering individual modules at an EQF level of 5 or above. 

A diversified (unbundled) landscape will likely require widely (internationally) recognised standards for 
provision and providers at each unbundled level, backed up by appropriate inspection and compliance 
bodies regulated by law. 

Recommendation 7: Address quality issues around credentialisation through qualifications 
frameworks 

Open education has recently led to an inexorable rise in ‘open’ or non-traditional qualifications. The same 
course, depending on various factors, might offer a learner learning badges, a certificate of attendance, 
a certificate of completion, a ‘verified’ certificate and/or a full qualification, translatable into university 
credit. A quality qualification should allow for the dual aim of: 

• certifying learning acquired (and the level of it where applicable) 

• facilitating the recognition of that learning for purposes of education and employment 


The Council of Europe considers recognition to be a “key element of the right to education and a 
responsibility of society”. The current expanding landscape of non-standard and/or institution-specific 
certification types, as outlined above, is significantly hindering the aim of recognition of that learning, in 
particular at global level, as well as creating considerable confusion among learners’ would-be learners 
as to the status of their credentials upon course completion. 

Global coordination of recognition needs to allow for a variety of qualification-types while at the same time 
keeping systems harmonised enough to allow for some level of standardisation. Actions for stakeholders 
to improve the recognition of open learning credentials might include: 

• At regional level, working with the hosts of the regional recognition conventions 19 , to address the 
emergence of open learning credentialisation through the coordinating structures created by each 
of the conventions. 

• At national level, working with recognition support offices to ensure they understand and ‘support’ 
open learning credentials. 

• At institutional level, incorporating the idea of ‘quality of credentials’ into the quality assurance 
systems, with a quality credential being one that certifies learning as accurately as possible, while 
facilitating recognition to the highest degree possible. 


Recommendation 8: Support knowledge transfer from open and distance learning to traditional 
quality systems 

Large scale online distance learning programmes have reached significant maturity in a number of 
countries, having already been active for over four decades. During this time, the institutions in question 
have gained significant experience in utilising technology to improve process management, course 


19 List of the conventions available from http://www.unesco.org/new/en/education/themes/strenqtheninq-education 
systems/hiqher-education/conventions-and-recommendations/ 
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content and student outcomes. In particular, the burgeoning field of learning analytics is providing insights 
into processes of teaching and learning, which were never before available for purposes of quality 
improvement. 

The experiences of open and distance learning institutions in implementing and using learning analytics, 
as well as other technically backed solutions for the enhancement of quality, has clear learning value for 
the rest of the education and quality assurance community. We therefore recommend that stakeholders 
should support a programme of best practice sharing between actors in the field, possibly through 
instruments such as master classes, knowledge sharing workshops and the like. 

Recommendation 9: Support quality assurance audits and benchmarking exercises in the field 
of online, open, flexible, e-learning and distance education 

There is a rapid development within the area, of open, flexible, e-learning and distance education. During 
this present study, stakeholders consulted have shown tremendous interested in this international quality 
research study. There have been suggestions that a quality assurance audit (such as this one) of the field 
on a 3-5 year cycle. The resulting reports could become the de facto reference point for quality assurance 
practices around the globe. 

This recommendation can be a natural input to the suggested quality Hub, in recommendation 3. 

It is proposed that ICDE should take on the responsibilities to carry out this task, as with ICDE’s 
international membership and reach render it is well placed to coordinate international reviews and 
ensure regular updating. Undertaking a regular cycle of audits can thus be a task for ICDE. Leadership 
of quality assurance audits would position ICDE as the meta-level resource keeping abreast of 
improvements in the field. 

Recommendation 10: Encourage, facilitate and support research and scholarship in the field of 
quality 

This study has highlighted the urgent needs for research in the field of open and online learning, including 
e-learning to match the speed in technological development. In addition there are also urgent needs for 
rapid dissemination and valorisation of research within the area, not at least to mainstream quality in e- 
learning. 

Implementation should address not just best practice, but also next practice and the needs for innovation 
and sustainability for the 21st century. 


Recommendation 11: Encourage, facilitate and support implementing quality assurance related 
to new modes of teaching 

There are urgent needs for stakeholders in the field to implement quality assurance related to new 
modes of teaching at the governmental, institutional and quality assurance agency level. 
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